1. Introduction¶
A real estate transaction can be an emotional time for everyone. The complexities between buyers and sellers are the result of different experiences and expectations. Success in today's market is guided by knowledge, communication, and partnership.
Buyers are waiting later in life to purchase their first home. They have very specific expecations on what they are looking for, and willing to take the time to get exactly what they want. To be successful, buyers will turn to experienced professionals to guide them through the buying process and to sift through the voluminous of data.
Sellers past experiences have been rooted in market conditions significantly different than we aree seeing today. Many are resisting the realities of the market and are slow to react to the valuable feedback the data provides. To be successful, sellers will need to utilize skilled professionals to interpret the specifics of today's market and take swift action to adjust for changing trends.
1.1 Problem Statement:¶
- How to predict a low risk / high yield return on property investment in a volatile market.
- Where and when to buy and sell that maximizes investment profits.
- Forecast future growth and decline of a region in order to guide investors with optimized, data driven, recommendations.
==========================================================================================
1.2 About the Data¶
Base Real Estate data provided by: Zillow
Base Federal Reserve data provided by: kaggle
Zillow Data: Timeseries Real Estate data by ZipCode U.S.
Zillow Home Value Index (ZHVI): A smoothed, seassonally adjusted measure of the median estimated home value across a given region and housing type. It is a dollar-denominated alternative to repeat-sales indices.
- Zip_Zhvi_SingleFamilyResidence.csv
- Zip_Zhvi_AllHomes.csv
- Zip_MedianRentalPricePerSqft_Sfr.csv
- Zip_MedianRentalPrice_AllHomes.csv
- Zip_MedianListingPrice_AllHomes.csv
Datahub.io: U.S., National Yearly Economic Reports
- interest_rates.csv
- Inflation, GDP deflator (annual %) and Inflation, consumer prices (annual %) for most countries in the world when it has been measured. Data The data comes from The World Bank (CPI), The World Bank (GDP) and is collected from 1973 to 2014. There are some values missing from data
- Inflation, GDP deflator (annual %) and Inflation, consumer prices (annual %) for most countries in the world when it has been measured. Data The data comes from The World Bank (CPI), The World Bank (GDP) and is collected from 1973 to 2014. There are some values missing from data
- inflation-consumer.csv
- inflation-gdp.csv
- education_budget_data.csv
- United States of America Education budget to GDP analysis Data Data comes from Office of Management and Budget, President’s Budget from white house official
- United States of America Education budget to GDP analysis Data Data comes from Office of Management and Budget, President’s Budget from white house official
- population.csv
- Population figures for countries, regions (e.g. Asia) and the world. Data comes originally from World Bank and has been converted into standard CSV
- Population figures for countries, regions (e.g. Asia) and the world. Data comes originally from World Bank and has been converted into standard CSV
- investor_flow_funds_monthly.csv
- Monthly net new cash flow by US investors into various mutual fund investment classes (equities, bonds etc). Statistics come from the Investment Company Institute (ICI)
- Monthly net new cash flow by US investors into various mutual fund investment classes (equities, bonds etc). Statistics come from the Investment Company Institute (ICI)
- housing_price_cities.csv
- Case-Shiller Index of US residential house prices. Data comes from S&P Case-Shiller data and includes both the national index and the indices for 20 metropolitan regions. The indices are created using a repeat-sales methodology.
- Case-Shiller Index of US residential house prices. Data comes from S&P Case-Shiller data and includes both the national index and the indices for 20 metropolitan regions. The indices are created using a repeat-sales methodology.
- household-income.csv
- Upper limits of annual incomes for each fifth and lower limit of income for top 5 percent of all housholds from 1967 to last year Data This dataset is acquired from U.S. Census Bureau, Current Population Survey, Annual Social and Economic Supplements.
- employment.csv
- US Employment and Unemployment rates since 1940. Official title: Employment status of the civilian noninstitutional population, 1940 to date from USA Bureau of Labor Statistics. Data Numbers are in thousands. US Employment and Unemployment rates since 1940 From the USA Bureau of Labor
- US Employment and Unemployment rates since 1940. Official title: Employment status of the civilian noninstitutional population, 1940 to date from USA Bureau of Labor Statistics. Data Numbers are in thousands. US Employment and Unemployment rates since 1940 From the USA Bureau of Labor
- cpi.csv
- Consumer Price Index for All Urban Consumers (CPI-U) from U.S. Department Of Labor Bureau of Labor Statistics. This is a monthly time series from January 1913. Values are U.S. city averages for all items and 1982-84=100.
- Consumer Price Index for All Urban Consumers (CPI-U) from U.S. Department Of Labor Bureau of Labor Statistics. This is a monthly time series from January 1913. Values are U.S. city averages for all items and 1982-84=100.
- cash-surp-def_csv.csv
- Repository of the data package of the Cash Surplus or Deficit, in percentage of GDP, from 1990 to 2013. Data Data comes originally from World Bank!
- Repository of the data package of the Cash Surplus or Deficit, in percentage of GDP, from 1990 to 2013. Data Data comes originally from World Bank!
- bonds_yields_10y.csv
- 10 year nominal yields on US government bonds from the Federal Reserve. The 10 year government bond yield is considered a standard indicator of long-term interest rates.
- 10 year nominal yields on US government bonds from the Federal Reserve. The 10 year government bond yield is considered a standard indicator of long-term interest rates.
- gdp_quarter.csv
- gdp_year.csv
- Gross Domestic Product (GDP) of the United States (US) both nominal and real on an annual and quarterly basis. Annual data is provided since 1930 and quarterly data since 1947. Both total GDP (levels) and annualized percentage change in GDP are provided.
- Gross Domestic Product (GDP) of the United States (US) both nominal and real on an annual and quarterly basis. Annual data is provided since 1930 and quarterly data since 1947. Both total GDP (levels) and annualized percentage change in GDP are provided.
Dataset Info: Economic
- The Time series data range our modeling and analysis was centered on was from 1997 through 2018. All of the Realestate datasets achieved this desired range, however some of the Economic datasets did not. To achieve paraty and have a fuller dataset for baseline testing, time series future forecast methods were applied. More will be described in section 2 on Time Series forecasting.
- GDP Yearly: Forecasted for 2016, 2017, 2018 values
- Inflation: Forecasted for 2017, 2018 values
Interest Rates: Forecasted for 2016, 2017, 2018
Note: Kaggel Federal Reserve datasets proved to be useless, full of gaps and limited time series data to provide value. Economic data was pulled from the above mentioned sources and munged together to form a more useable data set.
Dataset Info: Real Estate
- This data is our base datasets and provides the core insights into preditable housing market trends given prior knowledge of price performance coupled with economic fluctuations. Timeseries prediction models are created for each type of housing dataset mentioned above by ZipCode and it's monthly price value from 1997 to 2018. For this initial analysis, ZipCode's were focused to the U.S. State of Washington. This represents 351 unique zipcodes that were modeled with a five year future price prediction. These zipcodes then were combined with the economic features above, in order to create a dataset that could be used in identifing and or predicting events that could have a positive or negative impact on housing prices given a unique zipcode.
1.3 Obtain the data¶
- Using the base data available from Zillow
Zillow Home Value Index (ZHVI): A smoothed, seassonally adjusted measure of the median estimated home value across a given region and housing type. It is a dollar-denominated alternative to repeat-sales indices.
OBTAIN Interest Rates data from Kaggel
- Using the dataset provided by the kaggel Federal Reserve Interest Rates
- Using the dataset provided by the kaggel Federal Reserve Interest Rates
Obtain Economic Data from datahub.io)
1.4 Data Exploration - SCRUB - CLEAN - Transform¶
Clean and perform initial transformations steps of the data
REAL ESTATE DATATSETS - ZILLOW¶
- Rename 'Region Name' Column to ZipCode
- Convert ZipCode field to string
- Remove columns of non-interest:
- 'RegionID','SizeRank','City','Metro','CountyName'
- '1996-04','1996-05','1996-06','1996-07','1996-08','1996-09','1996-10','1996-11','1996-12'
- '2019-01','2019-02', '2019-03', '2019-04', '2019-05', '2019-06', '2019-07','2019-08','2019-09'
- 'RegionID','SizeRank','City','Metro','CountyName'
- Fill NaN with median value
Zillow Single Family Residence DataFrame Head:
INTEREST RATE DATASET - KAGGEL¶
- Datasets:
- Interest Rate:
- Rename column names to make it easier to work with
- View the new column names in a correlation heatmap
- Rename column names to make it easier to work with
- Interest Rate:
A look at the datasets distributions of elements to determin best methods for cleaning the data
ECONOMIC DATASETS - DATAHUB.IO¶
- Datasets:
- Interest Rate:
- keep Year, Month, Federal Funds Target Rate
- keep Year, Month, Federal Funds Target Rate
- Inflation Consumer:
- filter on Country = 'United States', keep Year, Inflation - drop the rest
- filter on Country = 'United States', keep Year, Inflation - drop the rest
- GDP Year:
- Change column names
- Change column names
- Interest Rate:
This process continuous for the remainder of the datasets. See accompaning notebook for details.
Merged Dataframe of Economic features aggregated from their individual source files
Correlation Heatmap of the new Economic Dataset's features
Impacts of Economic Factors on National Housing Price Index Avg¶
2. Time Series Analysis¶
Time series analysis on real estate median average price by zipcode
- Single Family Home Value
- Rental Price psf
- Listing Price Create future prediction models for all WA State zipcodes historical monthly housing price values.
2.1 Analysis¶
Transform Data¶
Transform Real Estate data for time series analysis
2.2 Exploration¶
2.3 Model¶
Note: All timeseries models were ran prior on google colab and saved as pickle files for continued downstream application
2.4 Results¶
Training Data Sets¶
Price Trend from 1997 through 2017 - With a 12 month future prediction...
Future Prediction Trends¶
Price Trend from 1997 through 2018 - With a 5 year future prediction...
Timeseries Models Performance Metrics - 90 days Horizon¶
Timeseries Models Cross Validation Metric Mape - 90 day Horizon**¶
3. Clustering¶
- K-means - unsupervised
- Mean-Shift - unsupervised
Description: Run k-means for three choices for k and choose the best.
A loop of 10 iterations were ran of the zipecode models generated from the Timeseries process ran above. Based on the output of the Elbow technique K=4 was the best chosen choose.
Intent: Try and use unsupervised learing techniques to classify Timeseries models produced by prophet.
Which are the best forecasters?
- Try and group into 3 classes using unsupervised learning
- Focus on single familey homes
3.1.1 Analysis¶
3.1.2 Exploration¶
Get all of the zip code forecast prediction models that generated in section 2 from disc, and prep for kmeans
Clean the forecast dataset for clustering¶
- limite the features for clustering - and the observations to just the predition time (5 years) + one year observed
- remove additive terms and multiplicative terms as well as the datetimestamp
- save series objects for later re joining
Pull in generated datasets for modeling...¶
- Economic Date - economic_forecast_date_norm.csv --- economic factors per date
- Real Esate Date - sfr_price_zip.csv --- single family homes price value per zipcode and date
3.1.3 Model - KMeans¶
- Run multiple k means to determin optimal k size for final model creation
- 8 iterations were ran, where k 4 was the most optimal
Build KMeans based on ideal cluster state found by Elbow method - 4¶
3.1.4 Results¶
Rusulting Cluster Classification at K equal 4
4. Decision Tree¶
- Decision Tree - supervised
- Include three different trees and their visualizations
Python package: scikit-learn sklearn.tree.DecisionTreeClassifier
*Build a decision tree model.
4.1 Analysis¶
- Transformation of the data's necessary to merge the datasets together after processed through prophet.
- Look over the distribution of key features
- Set price thresholds for supervised learning classification
- Price_Point_Class is a generated feature for supervised classification. Details are shown below
transform this data set to be in the shape: columns are zip codes, yhat is the value as prices, date¶
Final Merged Dataset - Real Estate Combined with Economic Data Features¶
Time range - 1997 - 2017 (that was the cleanest that could be achieved at this time...
*Train classifiers on Feature 'Price_Point_Class'
- 0: means observation's price value is < 25% of the State Price Average
- 1: means observations fall within the normal (average) range of the State Price Average
- 2: means observations falls above the 75% range of the State Price Average
--Determin if classifiers can identify future home value classes based on prior date, location and economic features that have the most impact on both postive and negative price value swings...
- Dataset Shape: (88452, 16)
4.2 Exploration¶
Look for imbalance in the sample observations for the target class¶
4.3 Model - DecisionTree Classifier¶
- max_depth: None (default)
- min_samples_split: 2
- randome_state: 42
4.4 Results¶
4.5 Random Forest Classifier¶
Python Package: scikit-learn v0.21.3 sklearn.ensemble.RandomForestClassifier
A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is always the same as the original input sample size but the samples are drawn with replacement if bootstrap=True (default).
- n_estimators: 100
- max_depth: None (default)
- min_samples_plit=2
4.5.2 Randome Forest Results¶
5. Naive Bayes¶
Python Package: SciKit-Learn Gaussian Naive Bayes
Build a naïve Bayes model. Tune the parameters, such as the discretization options, to compare results.
5.1 Analysis - Naive Bayes¶
5.2 Exploration - Naive Bayes¶
5.3 Model - Naive Bayes¶
- priors: None (default)
5.4 Results¶
6.1 Analysis¶
6.2 Exploration¶
6.3 Model - SVM¶
- Three rounds with different kernel's being evaluated
- 1st: rbf
- Results:
- Class1 - Best f1-score of .53
- Class1 - Best f1-score of .53
- Results:
- 2nd: poly
- Results:
- Class2 - Best f1-score of .57
- Class2 - Best f1-score of .57
- Results:
- 3rd: sigmoid
- Results:
- Class1 - Best f1-score of .53
- Class1 - Best f1-score of .53
- Results:
- 1st: rbf
6.4 Results¶
8. Final Results & Conclusion¶
Real estate housing market trends are impacted by many factors that require deep data mining techniques and domain experts to pull the right data together and engineer it in meaningful ways to gain insights into this industry. Data proved to be the most challening component of this research. There is a lack of quality datasets that are easily found which inhibits possible discoveries.
Certainly economic indicators are present that signal swings in price trends... Further research on comprehensive, state level economics is needed to expand on the datasets used in this study, which were at the national level. Most likely it's this that caused the inconsistencies with the models performance. The Real estate data being focused on was at the state level, whereas the economic data was at the national yearly average. This abstraction could have been a leading cause.